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L EXECUTIVE SUMMARY 



It was not long ago when workers could easily enter the job market. However, 
starting within recent decades, as the economy and businesses began adapting to 
advanced technology and expanding global markets, an imbalance developed between 
the skills and competencies students were acquiring and the skills and competencies 
businesses were demanding. This gap has continued to widen, impeding the 
att ainm ent of meaningful employment for workers and hindering the success and 
expansion of business. 

In the effort to find solutions to this problem and address the needs of students 
and workers, education and business, the U.S. Departments of Labor and Education 
and the Office of Personnel Management formed a partnership to develop assessment 
measures of workforce competencies and skills as they were defined by the 
Secretary's Commission on Achieving Necessary Skills (SCANS). The purpose of the 
assessment measures is to provide data about the skills of America's current 
workforce and about the skills individual workers need for our nation to aggressively 
compete in the global economy. However, because of the great consequences to 
individuals, and to the nation as a whole, it became essential to first know and 
understand the basic elements before the assessment tools were developed. What are 
the skills and competencies that workers need in order to secure and perform the 
complex jobs of today's and tomorrow's ever-changing economy? 

Sponsored by the U.S. Department of Labor, Employment and Training 
Administration, Office of Policy and Research and the U.S. Department of Education, 
National Center for Education Statistics, the National Job Analysis Study (NJAS) 
addresses this question by providing the research necessary to empirically identify 
and establish a taxonomy of workplace skills. 

It is generally accepted that with an ever-changing workplace, America's 
v/orkers must have the ability to move from one specific job with its unique demands 
to a totally different job with its demands. Critical for this new workforce are the 
"cross-occupational" skills. These are the skills that the NJAS must identify, those 
that are common across all occupations and necessary for worker and business 
success, particularly in high-performance settings. The outcomes of the NJAS will 
then guide the development of sound assessment instruments and establish the 
foundation for continuous and effective upgrading of educational and training 
curricula. 



Background 



The National Job Analysis Study originated from the work of the Secretary's 
Commission on Achieving Necessary Skills (SCANS), which had been the most recent 
large-scale project to identify cross-occupational competencies. When SCANS 
convened in 1990, its purpose was to determine the skills American workers need for 
job success in a global market and to create a plan of action to make sure those skills 
are developed by all citizens. The competencies and foundation, or basic, skills that 
resulted from SCANS were intended to set the agenda for what students would be 
taught in the nation's schools and how workers would be trained and retrained for 
the advanced jobs of the future. When ACT was awarded the contract to develop the 
assessment phase of the SCANS competencies, quantitative research became 
essential to validate the cross-occupational skills taxonomy and definition of high- 
performance workplaces. This research, in the form of the NJAS, ensures a sound 
basis for developing assessment instruments as well as the eventual informing of 
educational and training curricula. 

To establish this foundation of research, the NJAS is designed with two phases. 
Phase 1 of the project includes the administration and analysis of Survey 1 of the 
NJAS to determine an initial set of core behaviors and a concurrent, collaborative 
project to identify an initial set of characteristics of high-performance workplaces 
(HPWP). 

Phase 2 of the NJAS includes the administration and analysis of Survey 2, 
which will be comprised of the initial set of behavioral items determined by Survey 1 
analysis. From the outcomes of Survey 2, the Behaviorally-Anchored Rating Scales 
(BARS) instrumentation will be developed, which will directly inform the construction 
of the test blueprint for the assessment instruments. Phase 2 will also include the 
development of items to validate the test blueprint established through the BARS as 
well as the further development of a high-performance workplace instrument to 
establish the presence and identity of high-performance characteristics within 
business organizations. 



Phase 1 

The first phase of the NJAS was recently conducted and identified an initial set 
of core behaviors that are common across occupations. Using several established 
criteria, the process started with ACTs research team pulling task statements from 
databases containing occupational information and determining the statements' 
degree of commonality. These task statements were then translated into a common 
language and level of specificity, also according to set criteria. The translated 
behaviors as well as illustrative behaviors developed from the SCANS competencies 



and other NJAS-associated projects were compiled into two versions of 
Survey 1— Forms A and B— each containing half of the 214 survey items. In 
accordance with sampling criteria determined by experts in ACTs Research Division, 
the survey was distributed to 12,000 job incumbents in 6,000 organizations across the 
country. Over 160 occupations were sampled. The survey asked job incumbents to 
rate each behavior in terms of how important it is to the job and how frequently it 
is performed. The surveys were distributed using a comprehensive mailing plan, with 
postcard reminders and follow-up telephone calls to encourage participation. 

For the returned surveys, data analysis produced an overall rating of criticality 
for each behavior. Next, a panel of experts reviewed the results and determined a 
cut score, and ACT researchers selected out highly correlated items, determining the 
final set of behaviors most common and important across the occupations sampled. 
Because of the great importance and potential impact of these behaviors on education 
and workplace training, ACT will proceed with Phase 2 of the NJAS to further 
evaluate these behaviors prior to the start of developing the assessment instruments. 



Phase 2 

The purpose of Phase 2 is to cross-validate the initial set of core behaviors 
identified from the first survey, establish the relationship between the behaviors and 
job tenure, and set proficiency levels for the behaviors with associated knowledge, 
skills, and abilities. Covering the same set of occupations, ACT will send Survey 2 
to a new sample of 6,000 job incumbents in 3,000 organizations. Survey 2 will 
include the set of core behaviors identified from the first survey, the same importance 
and frequency rating scales, and m additional rating scale in which job incumbents 
will indicate when each behavior is needed on the job (e.g., at entry, after 6 months). 
This phase will generate preliminary data on the relationships among the behaviors, 
and data analysis will sort the core behaviors into similar groups. 

Content experts will then participate in a structured process to develop the 
Behaviorally-Anchored Rating Scales (BARS) (Smith & Kendall, 1963), including the 
corresponding proficiency levels and associated knowledge, skills, and abilities (KSAs) 
for each level. The BARS will form the basis for constructing the assessment 
instruments and will be employed to identify the levels of behaviors and KSAs that 
are necessary in high-performance workplaces. Once the instruments are validated 
in a representative cross-occupational sample study, they will be used to obtain an 
estimate of the degree to which the workforce has acquired the necessary behaviors, 
skills, knowledge, and abilities. The BARS system will also be useful in developing 
skill requirements for particular jobs, informing occupational classification initiatives, 
and guiding educators and trainers in curriculum development. 
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High-Performance Characteristics 



Essential to the identification of the behaviors and KSAs necessary in high- 
performance workplaces is the standard definition of "high performance/' It is agreed 
by business leaders, educators, and policymakers that high-performance business 
practices are a key element in the continued economic well-being of our country. 
However, a lack of broad empirical data on the specific characteristics of high- 
performance workplaces has limited organizations striving to make the transition. 

To meet the need for a conclusive definition of high-performance, an additional 
project is being carried out in conjunction with the NJAS. Conducted with 
collaboration from the National Alliance of Business (NAB) and the U.S. Department 
of Labor Office of the American Workplace (OAW), the first phase of this project 
identified an initial set of 36 high-performance characteristics. These characteristics 
will guide the further development of a common language and research of such 
workplaces and will form the basis for a new survey to establish the presence and 
identity of high performance in American businesses. During the cross-occupational 
validation of the BARS from the NJAS, ACT will develop and administer a high- 
performance workplace survey to be sent to business executives participating in the 
NJAS in order to further evaluate the 36 high-performance characteristics, identify 
workplaces that use "high-performance" practices, and relate those practices to the 
cross-occupational behaviors of the NJAS. With information from this study, 
organizations will have a guide to help them make the changes necessary for creating 
a high-performance environment. 



Outcomes 

Through empirical data from the NJAS and HPWP surveys, the work activities 
that are critical in the most competitive business environments in the nation will be 
identified. This will be the foundation for a system of job profiles, anchored by the 
behavioral information from the NJAS, to connect the demands of globally 
competitive jobs with the lessons taught to all learners. Specific occupational skills 
and standards will be linked to a series of broader sets of underlying skills (e.g., 
critical- thinking, problem-solving) that can be taught throughout the school years. 
By systematically linking these skills to the demands of our most competitive 
workplaces and incorporating them into a full educational experience, students and 
workers will begin to develop the necessary skills to become productive citizens in a 
global marketplace. 

Throughout the development and implementation of the NJAS, the SCANS 
competencies and foundation skills and the content of the Workplace Literacy Test 
(WLT) are evident. Therefore, the outcomes of the NJAS, BARS, and the HPWP offer 
an opportunity to review the contents of SCANS and the WLT in terms of their 
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importance and applicability to the workplace, particularly within high-performance 
workplaces. Specific NJAS behaviors have been cross- walked to the SCANS 
competencies and to the content of the WLT. In most cases, several NJAS behaviors 
were necessary to cover the broad content areas of SCANS and the WLT. Through 
analysis of these behaviors, a valid estimate is possible for each SCANS competency 
and foundation skill and each WLT-related skill. The BARS development will further 
the opportunity to review the taxonomy of SCANS and the WLT by establishing an 
empirically-based organizational structure for comparison to both SCANS and the 
WLT. The HPWP survey will then frame this review within the content of high- 
performance organizations. 

ACT is also working with the Department of Labor-funded Dictionary of 
Occupational Titles Reinvention Project (0*NET). By supplying the outcomes of the 
NJAS, ACT will be able to provide the 0*NET project with validated work skills and 
activities to ensure that a common language of occupational information is developed 
concurrently. A common language system will provide the vehicle for assisting new, 
transitional, or displaced workers in making effective job-match choices. When 
occupations and their requirements are described in the same language, it becomes 
viable to match the skills of the workforce with the job requirements, thereby making 
better employment decisions possible. 



ERIC 



5 



IL PHASE 1 OF THE NJAS 



To analyze the core job behaviors that ^11 form the basis for a workplace skills 
assessment, ACT has designed a plan based on the experience of job analysis experts 
both within and outside of ACT. The National Job Analysis Study makes efficient 
use of resident experts in job analysis, sampling, survey design, survey 
administration, data entry, data analysis, and psychometrics. To bring an even 
broader expertise to the project, ACT is also working with outside consultants and 
a panel of experts in the areas of job analysis, work skills, and education who play 
an active role in the development and review of materials. The NJAS is designed to 
ensure that the outcomes withstand the most stringent scientific scrutiny and meet 
the needs of the overall assessment project. 



Methodology and Development 

Considering the goals of the study, the first process of the NJAS was to select 
the methods for analyzing jobs, collecting data, and developing a representative 
sample of workers. This study requires a job analysis method that is feasible and 
suitable for the multiple uses of the results. There are at least four general 
approaches to job analysis: logical analysis, direct observation, task inventories, and 
the critical incident technique (Kane, Kingsbury, Colton, & Estes, 1986). Because of 
the large scope of this study, ACT selected logical analysis and a survey approach for 
the task inventory. 

The NJAS focus is on job, or work-oriented, behaviors. Once these behaviors are 
identified, it is feasible to determine the requisite knowledge, skills, abilities, etc. 
But in contrast to the more traditional task-based job analysis approach, the NJAS 
seeks to identify cross-occupational rather than job-specific work behaviors. As such, 
the study involves developing behavior statements that are generalizable across 
occupations. These common, or generic, behaviors are derived from task statements 
contained in job analysis information databases, from the SCANS competencies and 
foundation skills, and the Workplace Literacy Test. 

As a starting point, project staff conducted a search of ACT's RAVE (Realistic 
Assessment of Vocational Experiences) database, which contains information from the 
more than 12,000 jobs in the DOT updated through 1991. The information in the 
database was electronically queried on the basis of key words to locate task 
statements containing those words. The search began with the use of 157 action 
verbs identified by Gael (1983, p. 60) as ones frequently included in task statements. 
Other verbs were added as appropriate. 



Project staff also conducted an additional search in the database of job analysis 
information developed in 1992 for the Fort Worth, Texas, Independent School District 
and Chamber of Commerce. This database contains task statements from 
approximately 230 jobs in the Fort Worth area. These statements, which are coded 
by DOT codes, were compared to those contained in the RAVE database to ensure 
that the job analysis information for Survey 1 is as current and comprehensive as 
possible. Other databases, such as the one developed by the U.S. Office of Personnel 
Management (OPM), were also consulted to ensure comprehensiveness. 

After the common job task statements were identified, ACT staff reviewed the 
task statement clusters for meaning and level of specificity. ACT staff translated the 
statements into common, or generic, behaviors written in a common language and at 
an appropriate level of specificity. The iterative process developed by ACT for writing 
the behavior statements includes the following steps: 

• Group task statements by similarity of verbs. 

• Group task statements by similarity of verb objects. 

• Group task statements by similarity of outcomes. 

• Group task statements by similarity of knowledge or physical skills 
required. 

• Write behavior statements for task groups. 

After the sample task statement groups were translated, ACT staff reviewed the 
behavior statements for overlap, combining or eliminating statements as necessary. 
The statements were then edited to simplify the language as much as possible. 

The next step involved a meeting with ACT staff and the panel of job analysis 
experts. Using a modified Delphi technique, the meeting continued until a consensus 
was reached among all members of the panel as to the appropriate level of specificity 
in which to write all behavior statements (Gael, 1983; 1988). 

Once consensus was obtained, ACT staff developed and refined the behavior 
statements for all task groups, cor^ltmg experts in the corresponding occupations 
as needed. SCANS and other sources of generalized behaviors were compared to the 
content of the task-group behaviors. When gaps were discovered, the relevant 
illustrative behaviors were added. The panel of experts then reviewed the full list of 
behavior statements to ensure that they reflected commonality of language and the 
level of specificity agreed upon through the consensus process. The panel also 
evaluated the behavior statements for content and comprehensiveness, expression in 
behavioral terms, and reading level, which must be easily understandable for a vast 
majority of job incumbents. Behavior statements were revised or added as necessary 
before being compiled for Survey 1. 
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The final 214 behavior statements reflected the level of specificity determined 
by the panel of experts. These items were placed randomly on the two survey forms 
rather than grouped by statement type to ensure that each item was evaluated 
independently from similar items. A rating scale of importance and frequency was 
also included for each survey statement, and a blank section was provided so that job 
incumbents could list other general tasks they perform that were not included in the 
survey. Survey 1 also contained demographic statements on factors such as time on 
the job, job title, wage, gender, ethnicity, and age. The specific list was developed by 
ACT in conference with the project consultants and panel of experts. 



Sampling Frame 

Because the intent of this study is to identify behaviors that are common across 
jobs, the selection of occupations to be sampled is critical. It would have been 
prohibitive and unnecessary to include the more than 12,000 occupations covered in 
the DOT. However, the sample must be representative of the total set of occupations 
so that the results of the study are valid. 

ACT based the selection of a representative set of occupations on three criteria 
that, together, were most likely to yield an adequate sample from which to generalize 
to most occupations in the country. The three criteria are: 

• Commonality . ACT selected occupations that as a group employ 80 percent 
of workers as counted by the Occupational Employment Statistics (OES) system. 
This ensures that the behaviors identified in the study are performed by a 
substantial majority of the workforce. The sample should be large enough to 
provide stable estimates regarding the behaviors in question. 

• Representativeness Across Occupations . The sample for this study covers the 
full range of occupations regardless of the relative size of occupational families. 
ACT chose representativeness as a criterion to ensure that the sample was not 
restricted to a narrow set of occupations that employ large numbers of people. 
ACT used its "World-of-Work Map" (Prediger, 1976) to analyze the set of 
occupations selected by the first criterion and added occupations in areas that 
were proportionally underrepresented. 

• Growth . ACT selected occupations that are expected to grow most rapidly in 
the next decade. The sample included the 25 occupations with more than 
100,000 workers that are expected to experience the most real growth. When 
these occupations were not part of the sample derived from the first two criteria, 
they were added. 



ACT's selection process for the occupational sample is iterative. The first 
criterion generated 140 occupations. This set was compared to the job families on the 
"World-of-Work Map" and to the list of the 25 fastest-growing occupations. The 
resulting sample of 164 contained the top 25 growth occupations, and at least one 
occupation in each the 23 job families on the 'World-of-Work Map." This 
representativeness indicated that the set of occupations is well distil uted and, 
therefore, taps the breadth of skills that would represent all aspects of the libor 
market. 

Optimally, the sampling frame would take a random selection of people in these 
occupations, but, to ACT'S knowledge, no database of employees/occupations exists. 
Thus, it was necessary to contact the employees through the businesses at which they 
work. This was done by identifying the types of businesses at which it is likely that 
a person in the target occupation would be working. This was done using the 
crosswalk of the SIC (Standard Industrial Classification) codes and OES codes that 
are available through an Industry to Occupation (10) matrix. This crosswalk 
identified the types of industries to be sampled. To adjust for the fact that some 
businesses would be much larger than others, therefore giving different probabilities 
of selection for the people in the target occupation, ACT used the procedure known 
as Selection with Probability Proportional to Size. In this procedure, the businesses' 
chance for inclusion in the sample is proportional to how many employees they have. 
It should be noted that a slightly better selection method could have been to sample 
proportional to the number of people in the target occupation who work within a 
particular firm, however, this information was not available. Once the types of 
businesses were identified, ACT gathered the necessary information on the number, 
size, and mailing address of businesses from within the particular SIC code that was 
identified. In the information packet sent with Survey 1, ACT asked the identified 
business to select two people in the target occupation. The sampling within the type 
of business was implicitly stratified by region using the zip code. 

One expected difficulty from this method was that the identified business would 
not have a person in the targeted occupation. Anticipating this problem, the target 
business was given more than one job classification. ACT listed the target occupation 
at the top and below added four other occupations that a business in that SIC code 
should likely employ. The four other occupations were chosen randomly from those 
occupations that are common within that SIC code (and were to be included in the 
study). The instructions to the business then specifically stated that a person should 
be chosen from within the occupation listed at the top if possible, and, if there is no 
one from that occupation in the business, to select from the next one down on the list, 
etc. ACT understood beforehand that the matches between the type of firm (as given 
by SIC code) and occupation would be problematic; the above modification allowed a 
better chance of finding an employee in a targeted job classification. 
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Methods to Increase Response 

As with all surveys, the rate of response from the sample is extremely 
important. To ensure that sufficient numbers of respondents were available for 
reliable results, ACT used the following methods: 

♦ The surveys for Phase 1 were constructed so that they could be 
completed in a manageable time period. Based on past job analysis 
studies, ACT considered 30 minutes to be the maximum time burden on 
job incumbents. 

♦ Job incumbents should be motivated to complete the survey because 
they were being asked to participate by someone of higher authority in 
their organization and must return the survey to that person for mailing 
to ACT. To entrust confidentiality, each respondent would insert the 
completed survey into an envelope, seal it, and write his or her first 
name across the seal before returning it to the contact person. 

♦ ACT used a comprehensive mailing system that had been successful for 
other large-scale ACT survey projects. Follow-up telephone calls and 
postcard reminders were directed to nonrespondents, specifically the 
contact persons who were ultimately responsible for the return of 
surveys from their organizations. 



Survey Administration 

The first mailing letter announced the upcoming study and asked the contact 
person's organization to participate. This mailing also included a su mm ary 
describing the purpose and methodology of the study. Approximately two weeks 
later, the contact persons received survey packets for the job incumbent respondents 
and a letter further describing the study and outlining the procedures the contact 
persons should follow. The contact persons were responsible for distributing the 
survey packets to the incumbents and ensuring their return. The packets for 
incumbents included the survey and a letter explaining the importance of the study 
and expressing appreciation for their participation. The letter assured the 
confidentiality of the study and provided specific instructions for returning the 
completed survey to the contact person. The contact persons were provided a 
postage-paid envelope to return the surveys to ACT. 

After the survey packets were mailed, a postcard was sent to the contact 
persons who had not yet returned Survey 1. The postcard reminded the contact 
persons to promptly return the survey and reemphasized the importance of the study. 
Telephone calls were placed to nonrespondents urging them to participate and were 
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used to determine whether there would be any systematic bias that would affect the 
results of the study. Additional survey packets were sent to the contact persons who 
agreed over the phone to have a jacket resent. This survey packet also contained a 
letter restating the importance of the study and outlining the contact person's 
responsibilities for survey collection. 

In preparation for the returned surveys, quality control procedures were 
implemented at all steps of data entry and analyses. The data was entered into a 
large database at ACT and coded by OES codes. The data entry process was checked 
by ACT staff to verify the accuracy of this process. ACT staff implemented standard 
procedures for maintaining the confidentiality and security of all completed forms and 
other data. 



Survey 1 Outcomes 

Over 3,000 workers from 1,600 organizations took part in the initial survey. Of 
the occupations represented, secretaries returned the largest number of surveys (191), 
followed by registered nurses (115), college faculty (100), secondary school teachers 
(98), and bookkeeping/accounting clerks (92). Respondents also included engineers, 
postal carriers, salespersons, education administrators, librarians, bank tellers, auto 
mechanics, freight movers, machine operators, and police officers, among others. 

Descriptive statistics were computed on both the importance and frequency 
ratings for items in Survey 1. Criticality — a multiplicative composite index of 
importance x frequency — was calculated on the raw importance and raw frequency 
ratings. The means and standard deviations associated with this composite index 
were used to help determine the core set of behaviors across the jobs surveyed. 
Following analyses, all behaviors from Survey 1 were rank-ordered on the basis of 
their criticality. 

Results show that listening to the concerns of clients/customers and responding 
received the highest mark for the importance of the activity and the frequency it was 
performed (criticality mean [cm] 37.62). This activity was also listed by 82 percent 
of the respondents who perform this activity as part of their jobs. 

Also receiving attention were: the ability to use a computer to locate, process, 
or co mm unicate information (cm 36.19, 60%); safeguarding information and valuables 
(cm 36.15, 57%); scheduling work activities for oneself (cm 36.09, 80%); providing 
information to people (cm 34.98, 85%); determining the priority of work activities (cm 
34.88, 76%); working with people in other departments to accomplish work (cm 33.69, 
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85%); judging the importance, quality, and accuracy of information (cm 32.88, 70%); 
coordinating own work activities with the activities of others (cm 32.46, 81%); and 
listening to instructions from or concerns of supervisors or co-workers and responding 
(cm 31.98, 86%). 

These results are preliminary, pending validation in Phase 2. 



Outcome Analysis 

Through the deliberations of the panel of expert?, a qualitative method was 
established to identify the cut score for the "core" behaviors. Initial cuts clearly 
indicated behaviors to keep, those with high criticalities, and behaviors to drop, those 
with low criticalities. However, this left the median scores for interpretation and 
discussion by the panel of experts in order to establish a final cut score. Each member 
of the panel was asked to establish a cut score for the remaining behaviors. To be 
inclusive rather than exclusive, the experts considered the lowest cut score possible 
for including the behavior. Panel members met to discuss their decisions, and a 
consensus was reached. 

Attention was also given to the written comments of the job incumbents who 
completed the survey. Approximately 18 percent returned the survey with written 
comments. Of that 18%, approximately half concerned the respondents difficulty in 
interpreting the survey items, feeling that the work activities were too vague or not 
easily applicable to their specific occupation. Other noted comments felt that the 
items were repetitive and the survey too long for thoughtful completion within the 
estimated 30 minutes. 

Given that the purpose of the NJAS is to identify cross-occupational skills 
using common descriptions of the activities, and the fact that this is an unfamiliar 
purpose and survey, it was anticipated that incumbents would have some difficulty 
interpreting the survey items and their intent. However, ACT was concerned about 
the time burden and redundancy. Therefore, ACT felt it necessary to further analyze 
the survey data in the effort to trim the remaining items to lessen the time burden 
for the Phase 2 survey. No item limit was set, cuts were to be made based on 
analysis and consensus. 

From this point, ACT researchers correlated the items on each form. Because 
there were two forms, A and B, of Survey 1, this procedure could only analyze the 
items that were on the same form. Items with high correlations were identified for 
discussion with the options of keeping both items, editing the items to increase or 
decrease specificity in response to the correlation, or eliminating one item in favor of 
the other based on the correlation and other data. 
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ACT researchers then reviewed across forms for similar or redundant items 
based on one-to-one comparison and corresponding data. As in the above procedure, 
three options were considered. 

From the initial survey items, analysis identified the work activities that are 
considered the "core" behaviors. These final items are prepared for Phase 2 
validation. 
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IIL PHASE 2 OF THE N JAS 



The first step in Phase 2 is to develop a second survey consisting of the core 
behaviors identified in Phase 1. One key function of the second survey is to validate 
the outcomes of Survey 1. This cross-validation will help to ensure that these core 
behaviors are in fact critical in the workplace. Another function is to directly provide 
information for the development of the Behaviorally-Anchored Rating Scales (BARS). 

The format of Survey 2 will be modified to a small extent, including another 
rating scale. In addition to the importance and frequency scales used in Survey 1, 
Survey 2 will add a need scale for incumbents to assess when each behavior is needed 
on the job. This scale will provide valuable information concerning when the 
behaviors are important: at entry into a job, within the first year on the job, and one 
or more years after starting the job. The behaviors will then be sequenced on the 
basis of need, providing ACT staff with a logical approach for developing the BARS 
that target specific behaviors necessary at different points of time on a job. 



Outcomes of Survey 2 

The first analysis of Survey 2 will duplicate the procedures used with Survey 1. 
It is expected that the results at this phase in the NJAS will not reveal any 
prominent differences, thus validating the outcomes of Phase 1. If differences are 
evident, changes will be made accordingly to the final list of behaviors. The final 
analyses of Survey 2 will consist of procedures, such as cluster analysis, that will be 
used to group the behaviors into domains as the first step in developing an initial 
structure for the BARS. The analyses at this point are not intended as terminal, but 
are intended to offer an empirically based body of information for a committee of 
experts to form the final taxonomy. 

Development of BARS Instrumentation from NJAS Survey 2 Outcomes 

ACT researchers will take the core set of behaviors that result from Survey 2 
as the initial content for developing the BARS, which will provide a graphical method 
for scaling descriptions of behaviors along a hierarchical continuum ranging from low 
to high. This type of rating scale format is often used in taxonomic development to 
order content along a continuum and to provide more extensive operational 
definitions of domains. The BARS provide the methodology for scaling the behaviors 
from the NJAS Survey 2 according to complexity and to the knowledge, skills, and 
abilities (KSAs) required for the behavioral descriptions. 
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Subject matter experts will use the behaviors from Survey 2 to develop 
definitions for the anchor points on the scale. The anchor points for the domains will 
guide the experts in developing descriptions of proficiency levels based on the 
behaviors exhibited at the scale levels and the underlying KSAs. Accompanying 
skills-based literature will also assist experts in this process. Successful performance 
at a given proficiency level on the scale indicates mastery of the behaviors and KSAs 
associated with that level and with lower levels on the hierarchy, but not necessarily 
with higher levels of proficiency. 

There are 6 steps in the development of BARS for specific domains: 

Step 1: Organize the data into dimensions based on the NJAS outcomes 
and the Panel of Experts' judgements. 

Step 2: Scale the behaviors along a continuum on the basis of criteria 
such as complexity, level of details, and skill requirements. 

Step 3: Develop full level descriptions based on behaviors defining a 
level. 

Step 4: Identify vhe KSAs for each level description. 

Step 5: Pilot level and KSAs descriptions. 
Step 6: Validate the BARS. 

The outcomes of the BARS will inform the development of the test blueprint 
for the assessment instruments. 



Construction of the Test Blueprint 

Based on the BARS, relative weightings of dimensions can be assessed for 
construction of a test blueprint. This is accomplished by standardizing the mean 
criticality ratings of the behaviors within a dimension to determine their overall 
relative weighting as compared to other dimensions. From these standardized 
weightings, ACT can determine the proportion of measures to be constructed within 
dimensions to ensure adequate representation in the assessment. 

For example, a particular dimension may be divided into four level 
descriptions. Each level may contain five representative behaviors, established from 
the NJAS and the panel of expert's judgement. Based on the total criticality rating 
for all behaviors in the dimension, a percent weighting is calculated for each level 
description, reflecting the degree to which it is emphasized in the assessment. 
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IV. CHARACTERISTICS OF HIGH-PERFORMANCE WORKPLACES 



Also covered under the scope of the NJAS is a concurrent project designed to 
identify the characteristics of high-performance workplaces. Business leaders, 
educators, and policymakers agree that high-performance business practices are a key 
element in the continued economic well-being of our country. But without empirical 
data on the characteristics of high-performance workplaces, organizations have little 
to guide them in making the changes necessary for creating a high-performance 
environment and succeeding in a global market. 

To address this issue, the U.S. Department of Labor, Employment and 
Training Administration (ETA), and the U.S. Department of Education, National 
Center for Education Statistics (NCES), made funds available for a collaborative 
project to identify the key characteristics of high-performance work environments 
(HPWP). The project was conducted by American College Testing (ACT), the 
National Alliance of Business (NAB), and the U.S. Department of Labor Office of the 
American Workplace (OAW). A group of representatives from business, industry, and 
government identified a list of 36 characteristics they considered essential in their 
own definitions of high performance. This list was incorporated into an interview 
protocol that was administered to executives of 44 organizations (approximately half 
manufacturing and half service) considered to be high performance. The protocol 
asked executives to rate the 36 characteristics in terms of importance, and also to 
provide their own definitions of what high performance entailed in their own 
organizations. 



Outcomes 

From analysis of the importance ratings and key elements of the executives' 
own definitions, eight characteristics emerged that were considered most essential. 
These characteristics, which appeared in the top tiers of rankings but were not 
assigned a priority order, are: 

Focus on quality — The organization strives for continuous improvements in 
all processes, products, and services. Quality standards are built into product 
and service development and delivery systems. 

Focus on customer satisfaction/input — Customer feedback is valued and 
sought by the organization. Employees are focused on meeting customer 
needs. Customer recommendations about products or service improvements 
are incorporated into decisions made by the organization. 
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Flexible culture/openness to change ~ The organization views change as 
positive. Changes often reflect a quick and deliberate response to feedback 
from customers and employees. The organizational culture builds and 
reinforces the recognition that change is essential to the organization's renewal 
and vitality. 

Leadership with clear vision/commitment - Organization leaders convey 
through sustained action a vision of the company's purpose and value. Leaders 
articulate their vision for the future and provide a climate that engages 
employees in contributing to the vision. 

Training/continuous learning/education ~ The organization invests in and 
actively supports training and continuous learning at all levels of the 
organization. Education through classroom and on-the-job instruction is 
ongoing. A culture that values employee development exists within the 
organization. 

Information sharing - Information is shared quickly in the organization 
through top-down and bottom-up channels and across organizational 
levels/divisions using integrated information systems and technologies. The 
organization shares information, and employees' ideas and concerns are 
communicated throughout the organization. 

Profitability - The organization makes increasingly more money over time. 
Public/not-for-profit organizations work within budget constraints by being 
below budget or providing greater services. 

Productivity — The organization is committed to achieving productive outcomes 
through work-output-per-resource allocations that capitalize on employee input 
and technologies. 

Several of these characteristics also were mentioned most frequently in the 
definitions provided by the group of business, industry, and government 
representatives. Further research is needed to determine the significance of the 
differences between manufacturing and service organizations in rating the relative 
importance of particular high-performance characteristics. 

Therefore, these results should not be considered definitive. Further research 
will continue to arrive at & comprehensive definition of high performance. Hopefully, 
that research will address several issues that emerged from this collaborative study, 
such as whether the eight top characteristics carry more weight in implementation 
or are simply more common and therefore mentioned more frequently; whether some 
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characteristics are seen as drivers of high performance and others as supportive; and 
whether there are patterns or styles of high performance in which various 
characteristics are more commonly combined for specific organizations. 
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The results of this collaborative project will be used to inform development of 
an instrument for measuring high performance, which will be administered in 
conjunction with the NJ AS/BARS. In the future, organizations will be able to use the 
instrument to measure their own progress toward high performance. 

The first step in this process will involve an extensive literature review of the 
high-performance area to generate ideas and gather information before the actual 
survey development process begins. Experts in the field of high-performance (i.e., 
business, government, and academia) will then be consulted. Their task will be to 
elaborate on the 36 high-performance characteristics identified by ACT researchers 
in the above collaborative project and the literature review. Once the content 
domains have been determined, ACT will consult with experts to generate survey 
items. This final survey will be administered to the same organizations that complete 
the NJAS/BARS instruments. The results of the HPWP survey will be used to 
identify the characteristics associated with high-performance workplaces and to relate 
those to the results of the NJAS/BARS instrument data in these organizations. 
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V. ASSESSMENT DEVELOPMENT AND PILOT TESTING 



Assessment instruments are the final products of the NJAS. ACT will conduct 
a thorough electronic literature search and review to develop or choose workplace 
assessments based on the results of the NJAS/BARS. This literature analysis will 
include the traditional assessment literature. It will also include making contact 
with experts in the assessment field. This direct contact with experts will facilitate 
discovering new and emerging types of assessments that have not been published in 
the traditional assessment literature. Additionally, ACT researchers will conduct site 
visits to gather information about new and innovative types of assessment that are 
currently being used. These steps will help ACT establish communication between 
test development experts, and other professionals in the area so that the most 
current, valid, reliable, and cost effective assessments will be chosen or developed. 
To the extent that valid and reliable assessments are currently available, time and 
test development costs will be saved. 

The results of the literature analysis, telephone contacts, and site visits will 
form the preliminary basis for recommendations to the Departments of Labor and 
Education about assessment format and techniques. A panel of experts for the 
assessment phase of the project will review the recommendations prior to beginning 
assessment development. 

Assessment development begins with the creation of a test blueprint based on 
the results of the NJAS/BARS process. The blueprint will determine what percentage 
of items will be developed for each content domain. 

Once the test blueprint is established, ACT will proceed with item 
development. The finaliz d draft items will then be pilot- tested and submitted to the 
Departments of Labor and Education for review and approval, and after any revisions 
are made, an Office of Management and Budget (OMB) clearance package will be 
submitted. 

The steps in the assessment development process will begin with the 
identification of the type of items that would best estimate a person's level on each 
of the BARS instruments. The identification of item types will use the assessment 
literature analysis, ACTs experience in developing workplace assessments, and the 
combined experiences of the members of ACTs review committees. The next step will 
be to train item writers to write assessment items to measure each of the levels of 
the BARS. Once developed, the items will be edited and reviewed by content and 
testing experts. At this point, the items will be piloted by administering them to 
selected individuals. ACT will then interview those individuals to assess their 
understanding of the intent and content of what was being measured. Once reviewed 
and piloted, the items will be prepared for a large-scale field test and eventual 
external release. 
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VI. EXPANDED USES OF THE INFORMATION FROM THE NJAS 



The NJAS was initiated for the purpose of developing assessments. However, 
once the complete study is validated and evaluated, the NJAS information will be 
valuable to a number of other public and private initiatives whose goal is to equip 
American workers with the skills they and their organizations need to successfully 
compete. The common language of the NJAS/BARS makes it possible to compare and 
cluster occupations into broader areas and to develop profiles of the requisite 
behaviors, knowledge, skills, and abilities. These profiles can be used to facilitate 
worker transitions from one job to another, and to infuse workplace context into the 
K-12 educational system so that students successfully make the transition from 
school to work. These elements are the building blocks for an integrated system that 
prepares workers for challenging jobs in high-performance environments and for the 
lifelong learning those jobs require. 



Occupational Clusters and Skill Profiles 

One problem with occupational information to date has been the specificity of 
the language used to describe job content. In many cases, this information is only 
meaningful to the occupations described within a particular industry context. The 
lack of a standardized language not only makes it difficult to compare occupations 
across industry contexts, but it impedes efforts to train people in more general 
occupational areas and to help people make the sometimes necessary transitions from 
one occupation to another. The NJAS/BARS, with its common language acrorr, 
occupations and its emphasis on behaviors, knowledge, skills, and abilities, provides 
the mechanism for identifying similarities and differences among occupations across 
industries. This capability makes it possible to develop profiles of occupations or 
occupational clusters that include generalized work behaviors, specific job skills, and 
standards of performance. 

The clustering and profiling process begins with the proficiency levels and 
associated KSA's for the BARS developed from the behaviors in the NJAS taxonomy. 
Once proficiency levels and associated KSA's are established, it will be possible to 
begin developing skill profiles based on the behaviors and identified levels of 
proficiency required for various occupations and occupational areas. The skill profiles 
can be augmented by existing job-specific information so that they include both 
generakzable cross-occupational and occupation-specific information. 

In a sense, the descriptors will identify standards (in terms of levels) for each 
occupation or occupational cluster. Across industries, differences in the behaviors and 
KSA's of similar profiles can be noted. Further determinations can then ascertain 
whether the differences are due to the contextfindustry in which the occupation is 
performed or to true occupational differences. This will make it clear which behavior 
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and KSA's transfer across contexts and which are context-specific. The same applies 
to profiles within industries. Once skill profiles are determined, it will be possible 
to identify the common and unique elements that make up the profiles within 
industry contexts. 



School-to-Work 

To better ensure that this nation's economy is competitive in the future, the 
behaviors, knowledge, skills, and abilities (KSAs) that are most important for workers 
must be strengthened within the workforce. The first step in this direction is to 
identify where the most important behaviors and KSAs are being learned. The 
context for identifying the sources of learning will come from the BARS level 
descriptions containing the behaviors and KSAs in terms of where they are first 
learned and how they are effectively developed. This information will assist in 
identifying the most effective sites for learning the contents and exploring ways to 
better design efficient and effective work-related training and education programs. 

Identifying the sources of learning will rely on consultation with several 
business and industry trainers and teachers from various educational institutions. 
Through their involvement from both sides of the issues, their analysis and feedback 
on the proficiency levels associated with the BARS can help determine where the 
behaviors and KSAs are being taught and evaluated. Further discussions will 
ascertain possible changes to instraction/curricula and to the assessment process to 
ensure a greater level of skill transfer. 

The skill profile clustering system from the NJAS/BARS will also have direct 
relevance to school-to-work and work-to-work transitions. Through its use of a 
common language and implications for skill profile development and occupational 
classification, the NJAS/BARS can in fact link these initiatives into one system. 

The primary focus of the ctirricula development system must be to identify 
behaviors and KSA's that can be taught and used to guide the progress of young 
people and adults into meaningful and productive work. Typically, educators have 
taught very discrete types of skills that, although critical to learning and future job 
success, do not always transfer directly to the work context. To make this connection 
more direct, the NJAS/BARS system will describe profiles of behaviors, tasks, and 
KSA's at increasing levels of specificity, enabling teachers to discuss the workplace 
skills that every student needs upon leaving school, entering into training within 
general occupational clusters, or entering into occupations within specific industries. 
When students begin to make this logical progression in a real work setting, they can 
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build upon the levels and standards learned in the school. By maintaining the link 
between the general and specific behaviors/tasks and KSA's, students can begin to 
make transitions in efficient and predictable ways. They have learned the required 
elements not only at the specific job level, but also at levels more readily transferable 
to new learning and job contexts. 

Work-to-Work 

Because jobs change and people change jobs, the built-in linkages between the 
general behaviors and KSA's and their more job-specific counterparts can also allow 
easy transition from job to job. The ability to match a worker's current repertoire of 
behaviors and KSA's to those required for available or desired jobs greatly aids the 
worker's re-employment and the transition to a better job. Workers who meet the 
standards for other occupations can be placed in new jobs, and those who do not can 
be directed to appropriate training programs. 

The common language and organizational structure of the N J AS/BARS system 
offers the vehicle needed to match workers to jobs and to required job training. This 
information can form the basis for a labor market information system that matches 
workers to available jobs and to training programs. The result would be a matrix of 
occupational profiles linked using a common language to a current labor market 
database which can successfully match applicants to available jobs with similar 
profiles, highlight similarities over broad occuaptional categories, or indicate specific 
training to enhance the applicant's chances of a job match. 

A Comprehensive System 

The common language provided by the NJ AS/BARS system can link the school- 
to-work and the work-to-work initiatives into a single, comprehensive workforce 
learning system. This strategy unites the educational system (K-12) with the work 
context by infusing the clusters of behaviors and KSA's identified by the NJAS into 
school curricula. The common language of the NJAS/BARS connects occupations and 
occupational areas across and within industries to ensure certification and a high 
school diploma. Workers who meet the performance standards for an occupational 
area will be able to move from one industry to another because the behaviors and 
KSA's that are common across the clusters are known. Attainment of these 
transferable skills will increase the probability that the job of choice will be in a 
context best suited to the person's skill profile. 

This comprehensive strategy empowers learners with a profile of behaviors and 
KSA's necessary for entry into the workplace — specifically, high-performance 
environments — and for life-long learning in the work context. 
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